This application relates to automatic exposure control of digital cameras and other electronic digital image acquisition devices, and particularly to the control of scene illumination by flash light during the capture of image data of the scene.
Electronic cameras image scenes onto a two-dimensional sensor such as a charge-coupled-device (CCD), a complementary metal-on-silicon (CMOS) device or other type of light sensor. These devices include a large number of photo-detectors (typically two, three, four or more million) arranged across a small two dimensional surface that individually generate a signal proportional to the intensity of light or other optical radiation (including infrared and ultra-violet regions of the spectrum adjacent the visible light wavelengths) striking the element. These elements, forming pixels of an image, are typically scanned in a raster pattern to generate a serial stream of data representative of the intensity of radiation striking one sensor element after another as they are scanned. Color data are most commonly obtained by using photo-detectors that are sensitive to each of distinct color components (such as red, green and blue), alternately distributed across the sensor.
A popular form of such an electronic camera is a small hand-held digital camera that records data of a large number of picture frames either as still photograph “snapshots” or as sequences of frames forming a moving picture. A significant amount of image processing is typically performed on the data of each frame within the camera before storing on a removable non-volatile memory such as a magnetic tape cartridge, a flash memory card, a recordable optical disc or a removable hard disk drive. The processed data are typically displayed as a reduced resolution image on a liquid crystal display (LCD) device on the outside of the camera. The processed data are also typically compressed before storage in the non-volatile memory in order to reduce the amount of storage capacity that is taken by the data for each picture frame.
The data acquired by the image sensor are typically processed to compensate for imperfections of the camera and to generally improve the quality of the image obtainable from the data. The correction for any defective pixel photodetector elements of the sensor is one processing function. Another is white balance correction wherein the relative magnitudes of different pixels of the primary colors are set to represent white. This processing also includes de-mosaicing the individual pixel data to superimpose data from spatially separate monochromatic pixel detectors of the sensor to render superimposed multi-colored pixels in the image data. This de-mosaicing then makes it desirable to process the data to enhance and smooth edges of the image. Compensation of the image data for noise and variations of the camera optical system across the image and for variations among the sensor photodetectors is also typically performed within the camera. Other processing typically includes one or more of gamma correction, contrast stretching, chrominance filtering and the like.
Electronic cameras also nearly always include an automatic exposure control capability that sets the exposure time, size of its aperture opening and analog electronic gain of the sensor to result in the luminescence of the image or succession of images being at a certain level based upon calibrations for the sensor being used and user preferences. These exposure parameters are calculated in advance of the picture being taken, and then used to control the camera during acquisition of the image data. For a scene with a particular level of illumination, a decrease in the exposure time is made up by increasing the size of the aperture or the gain of the sensor, or both, in order to obtain the data within a certain luminescence range. An increased aperture results in an image with a reduced depth of field and increased optical blur, and increasing the gain causes the noise within the image to increase. Conversely, when the exposure time can be increased, such as when the scene is brightly lighted, the aperture and/or gain are reduced, which results in the image having a greater depth of field and/or reduced noise. In addition to analog gain being adjusted, or in place of it, the digital gain of an image is often adjusted after the data have been captured.
It is often difficult for the user to hold a camera by hand during an exposure without imparting some degree of shake or jitter, particularly when the camera is very small and light. As a result, the captured image may have a degree of overall motion blur that depends on the exposure time, the longer the time the more motion blur in the image. In addition, long exposures of a scene that is totally or partially moving can also result in motion blur in the captured image. A person or object moving across the scene, for example, may appear blurred in the image. The automatic exposure processing of existing cameras does not take into account motion of the camera or motion within the scene when calculating the exposure parameters to be used to capture an image of the scene.
U.S. patent application Ser. No. 11/258,975, filed Oct. 25, 2005, entitled “Camera Exposure Optimization Techniques That Take Camera and Scene Motion into Account,” does consider image motion when setting exposure parameters. Motion is detected and the exposure parameters are set, in advance of capturing data of the image, to levels that enhance the captured image based on the amount of motion of the scene relative to the image frame within the camera. Blur of the image caused by either camera shake or local motion within the scene, or both, can be minimized or even prevented by adjusting the exposure parameters. Conversely, in cases where little or no motion is detected prior to capturing the image data, the exposure parameters may be set to optimize other aspects of the image, such as increasing the exposure time in order to allow the depth of field to be increased and/or the level of noise to be reduced.
Motion is preferably measured by calculating motion quantities from data of two or more images prior to capturing data of the final image (using “pre-capture” images). Motion quantities that define the amount of motion of the scene image relative to the camera, including motion within the scene, are preferably calculated. Such relative motion quantities may include direction, thereby being motion vectors, or may just express the magnitude of the motion. By this technique, local motion vectors are individually calculated for distinct blocks of pixels within the image, which then allows motion within the scene to be taken into account when calculating the exposure parameters. Global motion vectors, such as caused by camera shake, can also be calculated from data of the two or more pre-capture images. Although the presence of motion blur can be detected from data of a single image, the calculation of motion vectors from two or more pre-capture images is more precise and leads to better control of the exposure parameters used to subsequently capture the image. Use of a mechanical motion sensor, which is included in some cameras, can only provide an indication of any global motion, not individual motion of objects or portions within the scene being photographed.
The results of the image motion calculations may also be used to estimate future motion so that a time to capture data of the image may be chosen where the absolute velocity of motion is at least less than at other times and possibly minimal. Particularly in the case of camera shake, where the motion often has some periodicity to it that can be forecasted, the picture can be taken at a time when the global motion is zero or near zero. The velocity of a portion of the scene can also be forecasted in the same way and a time chosen to take the picture when the local motion blur is minimized. When doing this forecasting, the exposure parameters are preferably calculated from the motion quantities that are expected to exist at the time scheduled for capturing the image.
In a specific implementation, when the ambient light is sufficient, preliminary exposure parameters are first calculated in the same manner as in existing cameras, without regard to any motion of the camera or portions of the scene image. If these preliminary parameters are at levels where their adjustment is not likely to improve the quality of the image, then the image is captured with them and the results of motion calculations are not used. An example where this can occur is with a brightly lighted scene, where the preliminary exposure time is nearly as short, the aperture nearly as small and the gain nearly as low as the camera allows. In such a case, the exposure time can neither be significantly shortened to limit any motion blur nor increased to significantly improve depth of field or reduce noise since the aperture and gain level are nearly as small as possible. But when this is not the case, the preliminary exposure parameters are adjusted on the basis of the image motion calculations to reduce the amount of or eliminate motion blur in the captured image.
According to improvements described herein, motion blur in the image may also be reduced or eliminated by controlling parameters of artificial light illuminating the captured scene. This is done both when a low light level of a portion or the entire scene makes it desirable to illuminate the scene with artificial light, or when the ambient illumination is sufficient but use of artificial illumination improves the quality of the captured image. In a specific implementation, motion blur of the resulting image may be reduced or eliminated by controlling parameters of the artificial light instead of adjusting the exposure parameters for this purpose. Such light is typically provided by one or more flash light sources, preferably built directly into the camera. Any or all of the intensity, duration, number and timing of flash light pulse(s) occurring during exposure may be controlled to reduce motion blur.
In yet another specific implementation, calculations of both the exposure parameters and those of artificial light that illuminates the scene may be made from the detected motion and these parameters then used to capture an image of the scene. The exposure and artificial light parameters cooperate to reduce or eliminate motion blur. They may also be selected to enhance the image by providing a more even luminance across it.
Further, when there is little or no motion of the image, the exposure duration and other parameters may at times be chosen according to the improved techniques herein to eliminate the need for artificial illumination or reduce its strength, thereby resulting in a captured image with better quality. But generally, the improved techniques described herein primarily allow acquiring images of scenes with low levels of ambient illumination that require artificial illumination. Parameters of the artificial illumination are calculated from quantities of motion detected in the image in order to reduce or eliminate motion blur.
The precise control of flash or other artificial light described herein is preferred over an approach of employing a sensor with greater sensitivity, and using the blur reducing techniques described in aforementioned U.S. patent application Ser. No. 11/258,975, without artificial illumination of the scene. This stems from the fact that increasing sensor sensitivity is becoming more difficult and expensive each year as the number of megapixels incorporated into the average image sensor used in a digital camera annually increases for competitive reasons. When the number of light gathering elements in a sensor increases, the size of each element is reduced, with a corresponding reduction in the sensitivity of each element, due to each element having a smaller area, and thus intercepting a lower amount of radiant energy.
The improved techniques of controlling exposure and/or artificial light parameters are also preferred over other ways that have been used or suggested for minimizing or eliminating motion blur in the image. An extremely short duration electronic strobe flash can be used to effectively stop any image motion so long as the exposure duration is limited to substantially the duration of the light pulse. But this also causes the resulting image to look very unnatural, with a brightly lit foreground and a very dark background. The control of exposure and/or flash parameters by detected image motion that are described herein allow better control of image brightness. For example, if the image motion is not extreme, the flash pulse may be made to have a longer duration than the short probe and thus a lesser intensity, thereby to provide better balance in image luminance between the largely flash illuminated foreground and ambient illuminated background.
Another approach made unnecessary by the improved techniques described herein is the use of optical stabilization to compensate for hand jitter that is provided on some cameras. In one line of cameras, vibration reduction lenses are used. A measurement of camera motion causes the position of the lens to be moved in a manner that moves the image in a direction and distance across the photosensor that is equal and opposite to the direction of image movement caused by motion of the camera. This is a complicated electromechanical system and cannot compensate for motion of one object within a scene relative to other objects of the scene.
Various aspects, advantages, features and embodiments of the present invention are included in the following description of exemplary examples thereof, which description should be taken in conjunction with the accompanying drawings.
All patents, patent applications, articles, other publications, documents and things referenced herein are hereby incorporated herein by this reference in their entirety for all purposes. To the extent of any inconsistency or conflict in the definition or use of terms between any of the incorporated publications, documents or things and the present application, those of the present application shall prevail.